orthogonally diagonalize matrix